Proceedings of the International Conference on Machine Learning 2024

In this paper, we propose QT -ViT models that improve the previous linear self-attention using quadratic Taylor expansion.

The objective is to predict potential energy and forces.

Wealso confirm these theoretical results experimentally. Further,we showempirically thattheKF-RTRLalgorithm captures long-term dependencies and almost matches the performance of TBPTT on real world tasks by trainingRecurrent Highway Networks on a synthetic string memorization task and onthe Penn TreeBank task, respectively.

However, there is currently no framework to apply it to generic architectures, specifically ones with linear weight-sharing layers.

In contrast, artificial agents are prone to'catastrophic forgetting' whereby performance on previous tasks deteriorates rapidly as new ones are acquired. This shortcoming has recently been addressed using methods that encourage parameters tostay close tothose used forprevious tasks.